fix(grafanactl): reconcile stale URLs and delete orphaned Grafana datasources by cssjr · Pull Request #258 · Azure/ARO-Tools

cssjr · 2026-06-25T21:35:07Z

Summary

The modify datasource reconcile command only managed Azure Monitor Workspace integrations (resource IDs on the Grafana ARM resource) but never checked the actual datasource URLs or removed orphaned datasources in Grafana.
After integration reconciliation, the command now:
- Updates stale datasource URLs when the AMW PrometheusQueryEndpoint hostname has changed (fixes DNS resolution errors)
- Deletes orphaned Managed_Prometheus_* datasources whose workspaces no longer exist (consolidates clean fixup-datasources logic into the pipeline step)
Orphan detection uses workspace existence, not endpoint presence, to avoid a delete-recreate loop with workspaces that exist but haven't provisioned a Prometheus endpoint yet.
Both operations respect --dry-run.

Fixes: ARO-27914, AROSLSRE-1347, AROSLSRE-585

Test plan

cd tools/grafanactl && go build ./... — compiles cleanly
cd tools/grafanactl && go vet ./... — no issues
Dry-run against dev Grafana — correctly identifies stale URLs and orphaned datasources
Live run against dev Grafana — 24 stale URLs updated, 10 orphaned datasources deleted
Verification re-run — 0 stale, 0 orphaned, 49 current (idempotent)
Verify dashboards still load after changes

🤖 Generated with Claude Code

Copilot

Pull request overview

This PR enhances grafanactl modify datasource reconcile so it not only reconciles Azure Monitor Workspace (AMW) integrations on the Managed Grafana ARM resource, but also detects and fixes stale Managed_Prometheus_* Prometheus datasource URLs in Grafana by aligning them with each workspace’s current PrometheusQueryEndpoint.

Changes:

Added a Grafana client helper to update an existing datasource via the Grafana API.
Wired a Grafana API client into the modify datasource reconcile command execution path.
Implemented datasource URL reconciliation logic for Managed_Prometheus_* Prometheus datasources, honoring --dry-run.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
tools/grafanactl/internal/grafana/client.go	Adds `UpdateDataSource` wrapper to support updating datasources via Grafana API.
tools/grafanactl/cmd/modify/options.go	Instantiates and carries a Grafana API client alongside existing ARM clients.
tools/grafanactl/cmd/modify/cmd.go	Collects AMW query endpoints and updates stale `Managed_Prometheus_*` datasource URLs (supports dry-run).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

The modify datasource reconcile command only managed Azure Monitor Workspace integrations (resource IDs) but never checked the actual datasource URLs in Grafana. When an AMW Prometheus query endpoint hostname changes, datasource URLs become stale and dashboards fail with DNS resolution errors. After integration reconciliation, the command now lists Grafana datasources, compares each Managed_Prometheus_* URL against the current AMW PrometheusQueryEndpoint, and updates any that differ. Fixes: ARO-27914 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…URL updates Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

geoberle · 2026-06-26T14:24:05Z

+			workspace.Properties.Metrics.PrometheusQueryEndpoint == nil {
+			continue
+		}
+		if *workspace.Properties.ProvisioningState == armmonitor.ProvisioningStateSucceeded {


workspaces can be in an another provision state (something something updating) intermittently

It now skips workspaces if they are Failed or Cancelled but includes workspaces in any other state (Succeeded, Updating, Creating). If a workspace is Creating but doesn't have an endpoint yet, it eventually gets skipped at the nil reference check.

…asources The modify datasource reconcile command only managed Azure Monitor Workspace integrations (resource IDs) but never checked the actual datasource URLs or removed orphaned datasources in Grafana. After integration reconciliation, the command now: - Updates datasource URLs when the AMW PrometheusQueryEndpoint has changed (fixes DNS resolution errors from stale hostnames) - Deletes orphaned Managed_Prometheus_* datasources whose workspaces no longer exist Both operations respect --dry-run. This consolidates the cleanup previously handled by the separate clean fixup-datasources command into the pipeline-integrated reconcile step. Fixes: ARO-27914 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…dpoint presence A workspace that exists but has no PrometheusQueryEndpoint yet would be kept as an integration (causing Grafana to maintain its datasource) but treated as orphaned by datasource reconciliation (causing deletion). This created a delete-recreate loop. Now orphan detection checks whether the workspace exists at all, and only skips URL comparison for workspaces without endpoints yet. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…tale-grafana-datasource-urls Adds orphaned datasource deletion and fixes workspace existence check to avoid delete-recreate loop. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Workspaces can be in transitional states like Creating or Updating intermittently. Only exclude workspaces in terminal failure states (Failed, Canceled) from integration and orphan detection, so that workspaces being updated are not temporarily removed and recreated. getWorkspaceEndpoints still requires Succeeded since transitional workspaces may not have a valid PrometheusQueryEndpoint yet. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

- Include datasource name and ID in UpdateDataSource error messages for easier troubleshooting - Collect all reconciliation errors (both deletes and updates) using errors.Join instead of returning on the first update failure, which would silently drop accumulated delete errors Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

A workspace transitioning through Updating still has its PrometheusQueryEndpoint from the previous Succeeded state. Use isTerminalFailureState instead of requiring Succeeded, and rely on the existing nil guard for workspaces that genuinely lack an endpoint. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

…r exist Orphan detection now includes all workspaces regardless of provisioning state. A workspace in Failed or Canceled state still exists as an Azure resource, so its datasource should be preserved — it may help identify broken clusters through the Grafana UI, and the pipeline will restore everything when the workspace is fixed. Datasources are only deleted when the workspace is truly gone (not returned by the API at all). Provisioning state filtering remains only for integration reconciliation and endpoint URL updates. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

raelga

/lgtm
/approve

janboll · 2026-06-29T09:38:12Z

/lgtm
/approve

janboll · 2026-06-29T09:40:01Z

/hold

cssjr marked this pull request as ready for review June 25, 2026 21:54

Copilot AI review requested due to automatic review settings June 25, 2026 21:54

Copilot started reviewing on behalf of cssjr June 25, 2026 21:55 View session

Copilot AI reviewed Jun 25, 2026

View reviewed changes

Comment thread tools/grafanactl/cmd/modify/cmd.go Outdated

cssjr force-pushed the fix/reconcile-stale-grafana-datasource-urls branch from fdc4ac3 to ba01314 Compare June 26, 2026 03:52

cssjr force-pushed the fix/reconcile-stale-grafana-datasource-urls branch from ba01314 to dc0e6a9 Compare June 26, 2026 03:59

fix(grafanactl): clarify dry-run vs live log messages for datasource …

0fb370b

…URL updates Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copilot AI review requested due to automatic review settings June 26, 2026 04:02

Copilot started reviewing on behalf of cssjr June 26, 2026 04:02 View session

Copilot AI reviewed Jun 26, 2026

View reviewed changes

Comment thread tools/grafanactl/internal/grafana/client.go

geoberle reviewed Jun 26, 2026

View reviewed changes

cssjr and others added 3 commits June 26, 2026 09:25

Merge branch 'fix/reconcile-grafana-datasources' into fix/reconcile-s…

d14b5f3

…tale-grafana-datasource-urls Adds orphaned datasource deletion and fixes workspace existence check to avoid delete-recreate loop. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

cssjr changed the title ~~fix(grafanactl): reconcile stale Grafana datasource URLs during modify~~ fix(grafanactl): reconcile stale URLs and delete orphaned Grafana datasources Jun 26, 2026

cssjr requested a review from Copilot June 26, 2026 18:56

Copilot started reviewing on behalf of cssjr June 26, 2026 18:57 View session

Copilot AI reviewed Jun 26, 2026

View reviewed changes

Comment thread tools/grafanactl/cmd/modify/cmd.go

Comment thread tools/grafanactl/cmd/modify/cmd.go

cssjr requested a review from Copilot June 26, 2026 19:01

Copilot started reviewing on behalf of cssjr June 26, 2026 19:02 View session

Copilot AI reviewed Jun 26, 2026

View reviewed changes

Comment thread tools/grafanactl/cmd/modify/cmd.go

Comment thread tools/grafanactl/internal/grafana/client.go

cssjr and others added 2 commits June 26, 2026 12:40

Copilot AI review requested due to automatic review settings June 26, 2026 19:45

Copilot started reviewing on behalf of cssjr June 26, 2026 19:45 View session

Copilot AI reviewed Jun 26, 2026

View reviewed changes

Comment thread tools/grafanactl/cmd/modify/cmd.go

cssjr mentioned this pull request Jun 26, 2026

feat(aro-hcp): add periodic Grafana datasource cleanup job (AROSLSRE-1138) openshift/release#80947

Closed

4 tasks

raelga approved these changes Jun 26, 2026

View reviewed changes

Uh oh!

Conversation

cssjr commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

geoberle Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

cssjr Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

raelga left a comment

Choose a reason for hiding this comment

Uh oh!

janboll commented Jun 29, 2026

Uh oh!

janboll commented Jun 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

cssjr commented Jun 25, 2026 •

edited

Loading